feat(agentx): bump Kimi-K3 FP4 MI355X ATOM image to 0924 and track recipe - #3407
Conversation
…cipe Track recipes/Agentic-Kimi-K3.md as retuned in ROCm/ATOM#2382: enable FlyDSL FP8 prefill attention on every band, and hold a ready prefill for four decode passes from concurrency 16 up. The published concurrency set and every other launch argument are unchanged. 将 MI355X Kimi-K3 FP4 ATOM AgentX 提交切换到 0924 镜像，并跟随 ROCm/ATOM#2382 重调后的 recipe：全部并发开启 FlyDSL FP8 prefill attention；并发 16 及以上时让就绪的 prefill 等待 4 个 decode 轮次。 已发布的并发点集合与其余启动参数保持不变。 Co-Authored-By: Claude Opus 5 <[email protected]>
gbyu-amd
added
AMD
agentx
|
Thanks for the contribution!
中文感谢你的贡献！
|
将 perf-changelog 条目的 pr-link 指向 PR 3407。 Co-Authored-By: Claude Opus 5 <[email protected]>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36493779111 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36493779111 |
…1613 Co-Authored-By: Claude Opus 4.6 <[email protected]>
|
InferenceX has switched away from unmaintainable bash scripts to YAML files that don't repeat the same stuff over and over again. Please merge the latest |
|
Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding |
…x-0924 Port the ATOM image bump and FlyDSL FP8 prefill attention onto the native srt-slurm recipe; the legacy script is deleted on main.
Carry srt-slurm patch 507 so ATOM aggregate workers accept extra-kv-connectors, and restore the DCP8 bands from the legacy config and ROCm/ATOM recipes/Agentic-Kimi-K3.md: concurrency 14 and 16 with DSpark 3 and ReplaySSM, 48, 56 and 72 without a draft, all on the in-process lmcache_offload connector (128 GB/rank, 192 GB/rank at 56 and 72). The PrefillDelayer applies from concurrency 16 up. The single-node adapter now reads ATOM's decode-context-parallel-size for the DCP_SIZE check, as it does for vLLM.
cquil11
added
full-sweep-enabled
|
/reuse-sweep-run 36493779111 |
…efill delay, DCP8 LMCache - Move kimik3-fp4-mi355x-atom-agentic-mtp to rocm/atom-dev:nightly_202609251613, tracking recipes/Agentic-Kimi-K3.md as retuned in ROCm/ATOM#2382. - Enable FlyDSL FP8 prefill attention (ATOM_USE_FLYDSL_FP8_PREFILL_ATTN=1) at every concurrency. - From conc 16 up, hold a ready prefill for four decode passes (ATOM_PREFILL_DECODE_INTERVAL=4, ATOM_PREFILL_DELAYER_MAX_QUEUE_MS=5000); conc 1, 4 and 14 unchanged. - Restore DCP8 LMCache bands on the native srt-slurm recipe: conc 14/16 (DSpark 3, ReplaySSM) and 48/56/72 (no draft) use ATOM's in-process lmcache_offload connector via roles.agg.args.extra-kv-connectors (srt-slurm patch 507), 128 GB/rank up to 48 and 192 GB/rank at 56/72. Conc 1 and 4 stay GPU-resident. Co-Authored-By: Claude Opus 5.5 <[email protected]>
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe at https://github.com/ROCm/ATOM/blob/main/recipes/Agentic-Kimi-K3.md
- No change to the Inferact/Kimi-K3-DSpark draft's precision: online_quant_config still excludes every draft linear (layers.*, context_proj), so its weights and activations stay BF16, and it keeps the target's FP8 KV cache (kv_cache_dtype fp8). FlyDSL FP8 prefill attention applies only to the target, since the draft runs its block pass as decode attention.
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj One thing blocks this: the sign-off still has no draft-precision evidence, which the checklist item requires. The additional detail section has not changed since the last verification. Every other check passes. Add the draft checkpoint, its stored precision, how the pinned image handles it by default, and the effective serving precision to the additional detail section, then re-verify. ❌ Check 13 (Draft runs as shipped): FAIL — Draft precision cannot be verified from the sign-off. It says only that the "datatype of the draft model is intact" because the srt-slurm commands did not change. That is not true: this PR bumps the image to Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep + evals on in-PR commit): PASS — In-PR commit ✅ Check 2 (Evals pass): PASS — The eval artifacts cover ➖ Check 3 (Recipe linked, merged, complete): N/A — This is a single-node ATOM recipe, and the recipe-link requirement only covers single-node vLLM/SGLang recipes. The sign-off links the ATOM recipe for reference. ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — Every item in the current PR_REVIEW_CHECKLIST.md template is present and checked. ✅ Check 6 (Upstream images / engine-first): PASS — (a) Does not apply: the framework is ✅ Check 7 (No deprecated models/scenarios): PASS — ✅ Check 8 (No architecture hacks): PASS — There are no ✅ Check 9 (Spec-decode via chat template): PASS — The recipe's benchmark env sets ✅ Check 10 (No engine patches): PASS — There are no patch files, source-rewriting heredocs or engine wheel installs. The srt-slurm patch 507 changes srtctl, not the engine, and is already on ✅ Check 11 (Agentic golden AL): PASS — ➖ Check 12 (Append-only): N/A — The new changelog entry does not set ✅ Check 14 (Pareto coverage): PASS — One curve: kimik3 / agentic-coding / MI355X ATOM fp4 / Assessed commit: |
…efill delay, DCP8 LMCache - Move kimik3-fp4-mi355x-atom-agentic-mtp to rocm/atom-dev:nightly_202609251613, tracking recipes/Agentic-Kimi-K3.md as retuned in ROCm/ATOM#2382. - Enable FlyDSL FP8 prefill attention (ATOM_USE_FLYDSL_FP8_PREFILL_ATTN=1) at every concurrency. - From conc 16 up, hold a ready prefill for four decode passes (ATOM_PREFILL_DECODE_INTERVAL=4, ATOM_PREFILL_DELAYER_MAX_QUEUE_MS=5000); conc 1, 4 and 14 unchanged. - Restore DCP8 LMCache bands on the native srt-slurm recipe: conc 14/16 (DSpark 3, ReplaySSM) and 48/56/72 (no draft) use ATOM's in-process lmcache_offload connector via roles.agg.args.extra-kv-connectors (srt-slurm patch 507), 128 GB/rank up to 48 and 192 GB/rank at 56/72. Conc 1 and 4 stay GPU-resident. - Draft model precision unchanged: online_quant_config still excludes every Inferact/Kimi-K3-DSpark linear (layers.*, context_proj), so its weights and activations stay BF16, and it shares the target's FP8 KV cache (kv_cache_dtype fp8). FlyDSL FP8 prefill attention applies only to the target; the draft's block pass runs as decode attention. Co-Authored-By: Claude Opus 5.5 <[email protected]>
seungrokj
left a comment
seungrokj
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe at https://github.com/ROCm/ATOM/blob/main/recipes/Agentic-Kimi-K3.md
- No change to the Inferact/Kimi-K3-DSpark draft's precision: online_quant_config still excludes every draft linear (layers.*, context_proj), so its weights and activations stay BF16, and it keeps the target's FP8 KV cache (kv_cache_dtype fp8). FlyDSL FP8 prefill attention applies only to the target, since the draft runs its block pass as decode attention.
Signed: seungrokj
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — in-PR commit ✅ Check 2 (Evals): PASS — ➖ Check 3 (Recipe link): N/A — this is an ATOM single-node recipe, not a vLLM/SGLang one, so the vLLM-recipes/SGLang-cookbook requirement does not apply. For reference, the linked ROCm/ATOM#2382 is MERGED (2026-09-24), and its FlyDSL FP8 prefill and PrefillDelayer (C≥16) settings match. ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — every current-template item is present and checked. ✅ Check 6 (Upstream images / engine-first): PASS — the only changed entry, ✅ Check 7 (Deprecated models): PASS — Kimi-K3 agentic coding with DSpark is an active scenario in ✅ Check 8 (No architecture hacks): PASS — no ✅ Check 9 (Spec-decode chat template): PASS — the recipe sets ✅ Check 10 (No engine patches): PASS — no ✅ Check 11 (Agentic golden AL): PASS — the ATOM adapter injects ➖ Check 12 (Append-only): N/A — the new changelog entry is not ✅ Check 13 (Draft as shipped): PASS — the ✅ Check 14 (Pareto coverage): PASS — run 36493779111 Assessed commit: |
cquil11
left a comment
cquil11
left a comment
There was a problem hiding this comment.
Full sweep 36493779111 green.
|
/reuse-sweep-run 36493779111 |
5 participants
Track the checked-in ATOM recipe
recipes/Agentic-Kimi-K3.mdas retuned in ROCm/ATOM#2382, on imagekimi_k3_agentic_0924.The published concurrency set [1, 4, 14, 16, 48, 56, 72] is unchanged; no points are added or dropped.
ATOM_USE_FLYDSL_FP8_PREFILL_ATTN=1on every band. ATOM defaults it to0, so the recipe's prefill attention path was not reached before this change.ATOM_PREFILL_DECODE_INTERVAL=4andATOM_PREFILL_DELAYER_MAX_QUEUE_MS=5000hold a ready prefill for four decode passes instead of interleaving it into every step. This is a threshold, not a band: concurrency 1, 4 and 14 run without it.max-num-seqs,max-num-batched-tokens,gpu-memory-utilization, the CUDA-graph ladder,dcp-size, draft depth, synthetic acceptance, ReplaySSM placement,AITER_REUSE_IDENTICAL_COMM_GROUPSand LMCache sizing are unchanged from #3207.Re-created on an in-repo
amd/branch so sweep dispatch and labels (AMD,agentx,full-sweep-enabled) apply.AI model disclosure
Prepared with Claude Code using
claude-opus-5(recipe reconciliation, edits, changelog entry). No other model contributed.Port to native srt-slurm and LMCache
Merged
mainin; single-node AgentX now runs on the native recipeinferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3/atom/mi355x-fp4-mtp/agentic.yaml, and the legacy script is gone.rocm/atom-dev:nightly_202609251613),ATOM_USE_FLYDSL_FP8_PREFILL_ATTN=1, and the PrefillDelayer from concurrency 16 up.recipes/Agentic-Kimi-K3.md: concurrency 14 and 16 (DSpark 3, ReplaySSM) and 48, 56 and 72 (no draft) on ATOM's in-processlmcache_offloadconnector, 128 GB/rank up to 48 and 192 GB/rank at 56 and 72, chunk size 1024,PYTHONHASHSEED=0. Concurrency 1 and 4 are unchanged.roles.agg.args.extra-kv-connectorsfrom srt-slurm patch507-lmcache-server-atom-sglang.patch(feat: support the LMCache server on ATOM and SGLang srt-slurm#32).decode-context-parallel-sizefor thedcp-sizecheck, as it already does for vLLM.